Skip to content

Promote staging to prod for EMR cutover #4535

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
sethstoudenmier merged 47 commits into from
Nov 5, 2025
Merged

Promote staging to prod for EMR cutover #4535

sethstoudenmier merged 47 commits into from
Nov 5, 2025

Conversation

@sethstoudenmier
Copy link
Contributor

@sethstoudenmier sethstoudenmier commented Nov 5, 2025

No description provided.

collinwr and others added 30 commits June 10, 2025 18:31
@collinwr
Overrides the QAT Spark bucket for EMR testing
e9113ce
@collinwr
Makes orphaned_award_info a unmanaged table
0bfaa09
@collinwr
Makes orphaned_transaction_info a unmanaged table
bdb8a56
@collinwr
Makes orphaned_transaction_info a unmanaged table (fixes f-string usage)
8134cdf
@collinwr
Adds IF EXISTS for dropping view to avoid errors
fb41314
@collinwr
Merge branch 'qat' of ssh://ssh.github.com:443/fedspendingtransparenc…
76e842b 
…y/usaspending-api into tst/emr-pipeline-testing
@sethstoudenmier
Change ES delta views to use CAST(... AS STRING) syntax
33cd6ab
@sethstoudenmier
Move away from subquery to support hive syntax
d414a65
@sethstoudenmier
Update unlinked awards download to temp tables in S3
882acac
@sethstoudenmier
Update covid download to use S3 location for readme
1de89b1
@sethstoudenmier
Clearing the faba table if it exists prior to cloning
c072e36
@boozallendanny
sbx testing
dce98a2
@boozallendanny
Merge branch 'tst/emr-pipeline-testing' of https://github.com/fedspen…
68758f9 
…dingtransparency/usaspending-api into tst/emr-pipeline-testing
@boozallendanny
data_broker_url
c0b5041
@boozallendanny
change data_broker_url to data_Broker_db
a529acd
@zachflanders-frb
testing removing subquery
11a2879
@sethstoudenmier
remove subquery in award_search incremental merge
1bab179
@boozallendanny
replace data_broker_db with broker_db
6263cad
@zachflanders-frb
Merge branch 'tst/emr-pipeline-testing' into dev-13454-transaction-se…
bed5252 
…arch-improvements
@zachflanders-frb
Merge branch 'tst/emr-pipeline-testing-no-subquery' into tst/emr-pipe…
15e6b99 
…line-testing-transaction-search-dataframe-integration
@zachflanders-frb
Update transaction search gold columns
2d92ba9
@zachflanders-frb
Update transaction search delta columns
cb067e0
@zachflanders-frb
Cleanup on transaction search delta columns dict
313a803
@zachflanders-frb
[DEV-13367] - Add schema for world cities
6edde0d
@boozallendanny
update env config for staging and prod
fca63da
@zachflanders-frb
Merge branch 'qat' into ftr/dev-13367-world-cities-schema
5572b38
@zachflanders-frb
Merge branch 'dev-13454-transaction-search-improvements' into tst/emr…
573605f 
…-pipeline-testing-transaction-search-dataframe-integration
@zachflanders-frb
Merge branch 'qat' into ftr/dev-13367-world-cities-schema
af62b8e
@zachflanders-frb
Merge pull request #4509 from fedspendingtransparency/ftr/dev-13367-w…
fc38f45 
…orld-cities-schema

[DEV-13367] - Add schema for world cities
@sethstoudenmier
handle merge conflicts
91f80f2
sethstoudenmier and others added 17 commits October 16, 2025 09:59
@sethstoudenmier
handle merge conflicts
3c4b446
@sethstoudenmier
Merge pull request #4518 from fedspendingtransparency/tst/emr-pipelin…
b91912b 
…e-testing-transaction-search-dataframe-integration

Combine the two EMR testing branches
@sethstoudenmier
formatting
cc98bf3
@sethstoudenmier
update path for COVID download README
5e8a994
@loreleitrimberger
[DEV-13689] update filtering when only a filter is provided, show chi…
d2d00dd 
…ldren
@boozallendanny
testing qat config with spark bucket
c5407e0
@loreleitrimberger
Merge pull request #4522 from fedspendingtransparency/ftr/dev-13689-u…
968c853 
…pdate-filtering-result

[DEV-13689] update filtering when only a filter is provided, show chi…
@sethstoudenmier
[DEV-13722] Truncate the File C update table instead of recreating
0be948a
@sethstoudenmier
Merge pull request #4523 from fedspendingtransparency/mod/dev-13722-u…
3c0a09f 
…pdate-truncate-file-c-table

[DEV-13722] Truncate the File C update table instead of recreating
@sethstoudenmier
Merge branch 'staging' of https://github.com/fedspendingtransparency/…
da3e1e9 
…usaspending-api into tst/emr-pipeline-testing
@sethstoudenmier
[DEV-13706] Use varchar for support between Postgres and Hive; add ne…
01c300a 
…w table_spec for lookup tables
@sethstoudenmier
[DEV-13706] Update references to old Broker DB env
1949891
@sethstoudenmier
[DEV-13706] Add back test data
521cf41
@sethstoudenmier
[DEV-13706] Some cleanup
fe45b02
@sethstoudenmier
Merge pull request #4532 from fedspendingtransparency/tmp/qat-confif
0aeca6c 
Updating the QAT config
@sethstoudenmier
Revert change to qat s3 bucket name
1e13c9a
@boozallendanny
Merge pull request #4519 from fedspendingtransparency/tst/emr-pipelin…
2cd0544 
…e-testing

Tst/emr pipeline testing
@sethstoudenmier sethstoudenmier added the env promotion [PR] promotes changes in a lower environment to a higher environment label Nov 5, 2025
@sethstoudenmier sethstoudenmier merged commit e61e4f5 into master Nov 5, 2025
54 of 56 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@boozallendanny boozallendanny boozallendanny approved these changes

Assignees

@sethstoudenmier sethstoudenmier

@boozallendanny boozallendanny

Labels

env promotion [PR] promotes changes in a lower environment to a higher environment

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@sethstoudenmier @boozallendanny @collinwr @zachflanders-frb @loreleitrimberger